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PROCESS FOR ASSOCIATING AND DELIVERING 
DATA WITH VISUAL MEDIA 

RELATED APPLICATION 

This application claims priority from provisional application Serial No. 
60/224,459 filed August 10, 2000. 

BACKGROUND OF THE INVENTION 

The present invention generally relates to distributing broadband 
content and data. More particularly, the present invention relates to a 
process for associating and delivering data with visual media, and has 
particular application to associating audio and description narration with 
visual media for the benefit of the severely visually impaired. 

According to United States census data, thirty-one million people in 
the United States are unable to completely enjoy movies or television 
because of severe visual impairment. Although the visually impaired can 
listen to the dialog between the various actors, as well as sound effects and 
music, they are unable to ascertain aspects of the film which are not spoken 
such as the background setting, character dress, relational placement of the 
characters, and unspoken action. It is estimated that the average movie 
contains forty-five minutes of unspoken action. Thus, a visually impaired 
person is literally left in the dark as to what is happening during the movie 
during these forty-five minutes. 

Recently, the Federal Communications Commission has mandated 
television and cable networks begin offering "audio description" which would 
describe the unspoken action and other necessary narrative elements. 
According to the mandate, the television and cable producers must do so 
through the secondary language (SAP) channels on televisions. However, 
the vast majority of television and cable stations are not currently equipped 
with SAP systems. This will require an enormous financial investment on the 
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television and cable producers part to obtain the appropriate SAP analog 
equipment. Furthermore, such SAP systems require appropriate 
engineering, constant maintenance by qualified video engineers, and 
enormous storage space as the equipment must be air conditioned. Such 
equipment will become obsolete in a few years when the television and cable 
industry completely converts to digital. The cable industry association 
estimates that small cable companies alone will have to spend over 20 
million dollars, and the entire industry close to 1 billion dollars to comply with 
the FCC ruling. 

Accordingly, there is a need for a process for associating audio 
description within visual media, such as television and cable programming, 
which does not require television and cable stations to acquire SAP systems 
and equipment. What is further needed is a process for associating encoded 
audio description within the visual media so that only those wishing to listen 
to the audio description can do so selectively. Such coded audio description 
should not interfere with the presentation of the visual media. The present 
invention fulfills these needs and provides other related advantages. 

SUMMARY OF THE INVENTION 

The present invention resides in a process for associating and 
delivering data with a video signal. The general steps of the process 
comprise first encoding a video source signal by inserting data in unused 
video bandwidth of the video source signal. The encoded video source 
signal is then transmitted to its destination, where it is decoded. The data is 
separated during the decoding process and either visually displayed or 
audibly delivered to an end user. 

The encoding step includes the step of digitizing an analog data 
signal. Typically, the analog signal comprises an audio signal. In a 
particularly preferred form of the invention, the audio signal comprises an 
audio narrative description of visual media associated with the video source 
signal for the benefit of the visually impaired. The digitized data is then 
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compressed and transcoded for insertion into predetermined unused video 
lines of the video source signal. 

The decoding step includes the steps of decompressing the inserted 
data after it is separated from the video source signal. The decompressed 
data is then converted from a digital format into an analog signal. When the 
analog signal comprises an audio signal, this signal is delivered to audio 
speakers, such as a headset worn by a visually impaired person. 

As the data is associated with the video signal so as not to interrupt 
the transmission and reception of the video signal, the unused bandwidth of 
the video signal can be advantageously used to convey additional 
information. This may include a narrative description of the visual media so 
that a visually impaired person can be informed of the background setting, 
character dress, relational placement of the characters, and unspoken action 
of the visual media. This narrative description could also comprise on-screen 
visual messages, such as television program guides, and the emergency 
broadcast system visual messages. Of course, the invention is not limited 
to these uses, but can have other applications in which data can be 
advantageously associated with a video signal in a transparent fashion. 

Other features and advantages of the present invention will become 
apparent from the following more detailed description, taken in conjunction 
with the accompanying drawings which illustrate, by way of example, the 
principles of the invention. 

BRIEF DESCRIPTION OF THE DRAWING 

The accompanying drawings illustrate the invention. In such 
drawings: 

FIGURE 1 is a flowchart illustrating the steps of encoding a video 
source signal in accordance with the present invention; 

FIGURE 2 is a flowchart illustrating the steps taken in decoding the 
video source to remove the inserted data for delivery in accordance with the 
present invention; 
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FIGURE 3 is a representation of a video screen, illustrating lines of 
visible video, and lines which are reserved or unused and not typically 
viewed; 

FIGURE 4 is a schematic block diagram illustrating the process of 
encoding an original video master in accordance with the present invention; 

FIGURE 5 is a schematic block diagram illustrating the process of 
decoding the video master and separating the video signal and inserted 
audio data in accordance with the present invention; 

FIGURE 6 is a schematic block diagram illustrating the general 
process of encoding an original video master with an audio narrative 
description for the benefit of the visually impaired in accordance with the 
present invention; and 

FIGURE 7 is a schematic block diagram illustrating the process of 
encoding the original video master of FIG. 6. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

As illustrated in the accompanied drawings for purposes of 
illustration, the present invention is concerned with a process for associating 
and delivering data with a video signal. With reference to FIG. 1, data 
content to be inserted into a video signal (1 0), which may comprise either an 
audio signal (12) or visual graphics (14) or a combination thereof is provided. 
This content is typically in an analog signal format originally. In order to be 
associated and delivered with the video signal, the content is digitized (16). 
The digitized content is then compressed (18) and the compressed signal 
transcoded (20). The compressed and transcoded data content signal is 
then mixed with the video source signal (22), which is sent to the output 
medium (24), such as a video tape, DVD, MPEG file, digital tape, etc. The 
foregoing steps illustrated in FIG. 1 are collectively referred to as the step of 
encoding the video source signal with the data content. 

The encoded video source signal is then transmitted to the end user, 
such as by transmitting through an Internet connection, playing a video tape 
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in a VCR, a DVD in a DVD player, or by cable or television transmission or 
the like. Due to the formatting of the data content, it essentially becomes one 
with the video source signal so as to be effectively transparent to all existing 
broadcast systems, equipment, players, etc. 

With reference now to FIG. 2, the mixed and encoded source signal 
(26) passes through a decoding circuitry where it is decoded (28). The 
inserted data content is then separated and extracted (30) where it is 
displayed on screen, such as over non-encoded video, if the data content 
comprises graphics (32) or is routed to audio speakers or headphones in the 
case of audio data content (34). 

It has been found that broadband content signals, particularly video 
signals such as television signals, have unused bandwidth, lines or holes 
which can be advantageously used to transmit data. So long as the unused 
bandwidth can be determined and found, and the data content to be 
distributed formatted appropriately to be fit within the unused bandwidth, 
simultaneous transmission is possible. The invention can have applicability 
to Internet, videos and graphics, distribution of information across wireless 
networks, information conveyed to personal digital assistance and other 
hand-heid electronic devices, etc. The present invention is particularly 
adapted to be used on video input to televisions, and television broadcasts. 

With reference to FIG. 3, a representative diagram of a televison 
screen (100) is shown. According to the NTSC SMPTE specification, there 
are a total of 525 video lines available for use. However, only 480 of those 
lines (1 02) are actually used for visible video. The remaining lines (1 04) are 
either unused or reserved for such things as closed captioning, picture 
control, etc. These lines are typically not viewed on a television set due to 
the extension of the frame of the television box over these unused lines. 
Different formats, such as PAL used in Europe, similarly have unused or 
reserved video lines of unused bandwidth. The present invention utilizes 
these unused lines or bandwidth to its advantage in order to associate and 
deliver data content which might otherwise be incompatible, or unrelated to 
the video signal transmission. The invention does this in a manner which 
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renders the associated and inserted data invisible or transparent to existing 
broadcast systems and equipment so that an end user desiring to extract the 
inserted data content need only have a decoder. Those not having a 
decoder would not be aware that additional information is contained within 
the unused bandwidth. 

With reference now to FIG. 4, a schematic diagram of a recorder for 
a television application is illustrated. An original master video tape, CD, or 
DVD, or like is played in a system (106) where the video and audio signals 
(1 08) are separated. The audio or data content in analog signal is converted 
to digital (110) by appropriate circuitry. This digital signal is compressed 
(112) and then synchronized (114) with the video signal. This mixed and 
synchronized video signal having the data content inserted therein is re- 
recorded at track recorder (116) to overlay the original audio signal. This is 
recorded (118) on a new master (120) video tape, CD, or DVD. 

Referring now to FIG. 5, the new master (120) is transmitted via 
cable, television broadcast, or played in the appropriate player, such as a 
VCR tape player, DVD player, or the like. A decoder, in the form of a set-top 
box, or electronic circuitry built into the television or player system includes 
a signal separator circuit (122) that extracts the inserted data content from 
the movie signal which is sent to the display device such as a T.V. (124). 
The extracted data content is then decompressed (126) using appropriate 
circuitry, after which it is converted back into its original analog signal format 
(128). At this point, the analog signal can be sent to an amplifier (130) for 
transmission to an audio speaker (1 32) or headset directly connected to the 
amplifier via a-jack or the like. Alternatively, the analog signal can be 
transmitted to a radio frequency transmitter (134) or transmission to an 
antenna (136) for wireless speakers, or a wireless headset. The systems 
shown in FIGS. 4 and 5 assume that the inserted data content comprises an 
audio signal. However, it is to be understood that the invention is not limited 
to such, and can include graphics which are overlayed on the television or 
monitor or the visual medium of the video source signal. Such graphics 
could also be separated and sent to a separate monitor or television set. 
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The process of the present invention allows "audio description 
narration" of a visual media to be encoded permanently onto the show picture 
master, thereby locking forever, regardless of whether the picture is copied, 
edited, or rebroadcast. Such audio descriptions are prepared prior to 
creation of the show picture master. Such audio descriptions will incorporate 
a narrative of unspoken action, or other necessary background information, 
which is seen but not heard in the visual media. Such visual media can 
include film, movies, television programming, and the like. 

Referring to FIGS. 6 and 7, the original source master (1 42) includes 
both original video and audio signals (144) and (146). The video signal (144) 
is fed into and encoder (148) where it is processed, typically at a CCIR 601 
level. Simultaneously, the audio signal (146) is fed into the encoder (148) in 
an unprocessed manner. A signal (150) from an audio description narration 
master (152) is also fed into the encoder (148). A closed captioning signal 
(1 54) may also be fed into the encoder (148) so that a new video signal (1 56) 
including the original video source signal (144), the inserted data from the 
description narration audio signal (150), as well as the closed captioning 
signal (154) is produced and saved on a new picture master (158). The 
original audio signal (146) is overlaid, and if necessary, re-synchronized, with 
the mixed and encoded video signal (156). 

The encoder (148) is designed to accept both video and audio inputs 
for processing. The encoder (148) can function both as a dedicated 
hardware device or software application within editing system modified for 
non-linear video editing, such as AVID, Adobe Premier, After Effects, Final 
Cut Professional, and the like. The video source inputs can include 
composite, component, serial digital, DVD, MPEG, and all streaming formats. 
The audio source inputs can include composite, digital, analog XLR balanced 
and unbalanced SPDIF (Sony Phillips Digital) and streaming. 

The encoding process takes a narrative audio sample of 
approximately 8KHZ in band width and converts that analog signal into a 
digital data stream. The data is further encoded and recorded to fit onto a 
single unused line of NTSC or PAL video. 
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Referring now to FIG. 7, a more detailed schematic illustration of the 
encoder (148) is shown. As stated above, the audio signal from the source 
tape (146) is fed into the encoder (148) where it typically is simply passed 
through unprocessed so as to remain 1 volt peak to peak, or re-synchronized 
if necessary to the record deck (1 60) where it is overlaid with the mixed video 
signal (156). 

The video signal (144) from the source tape is fed into the encoder 
(148) through a video interface (162), where it may be necessarily decoded, 
before being fed into a video record mixer (164). Simultaneously, the audio 
signal (1 50) from the narration master (1 52) is fed into the encoder (148) so 
that the audio analog signal is converted to a digital stream by A/D converter 
(166). A codec receives the digital audio signal and processes it to remove 
unnecessary data in order to compress and reduce the size of the digital file. 
Several companies have specific codecs for 8K audio such as Qualcomm, 
Motorola, Quicktime and MP3, and RealPlayer. The remaining compressed 
digital audio file is sent to a transcoder (170) which inserts edit in the video 
record mixer (164) on a single line of video. The transcoder translates the 
language of the incoming signal into the language of the target signal or 
medium. This involves synchronizing and conforming voltages, bandwidth, 
bit rate, etc. so that the processed signal (172) is compatible with the video 
signal (144) Depending upon whether the signal is to be produced in the 
United States or abroad, the single line of video is inserted as NTSC or PAL. 
The narrated audio file is compressed to fit in a 32 KB band width in order to 
fit within a single line of video reserved for picture control, closed captioning 
and the like. The digital transcoded narrative audio signal (172) is inserted 
into one of these reserved lines. Such reserved lines of video do not 
interfere with the appearance of the broadcasted visual media, but rather are 
hid, for example, within the boxed portion of a television set. Furthermore, 
these reserved lines of video are transparent to the broadcaster equipment. 

The video record mixer (164) combines the video signal (144) with 
the transcoded signal of the narration (1 72) and sends this signal (1 56) to the 
record deck (160). The video signal (144) that came from the original master 
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(142) now has a single line of video digital audio recorded on a chosen line 
and is recorded along with unprocessed audio (146) from the original master 
for recording on a tape or digi-beta tape which becomes a new picture 
master (158). Closed captioning (1 54) can interface with the encoder (148) 
to allow the dual encoding of both the closed captioning and narrated audio 
into the video signal (144) at the video record mixer (164) so that the closed 
captioning (154) is included on the new picture master (158). The closed 
captioning (154) is digitally set for a different line than the narrated audio 
(150), but both can be combined at the same time. 

By encoding the show masters of all broadcast programming, similar 
to closed captioning, the audio narration can pass transparently through all 
existing broadcast systems and equipment. 

The end users, the visually impaired and blind, will hear the 
associated audio description by one of several different means. Existing 
televisions can incorporate a decoder box to play the audio through the 
speakers of the television set. Alternatively, this signal can be sent directly 
to a head set worn by the visually impaired end user. The use of a headset 
allows those having normal sight to view the broadcasted programming in 
normal fashion without the audio description. It is anticipated that newly 
produced television sets will contain a decoder chip set which will take the 
line of video and produce the audio description for play directly through the 
speakers of the television. Of course, the signal can alternatively be sent to 
a head set worn by the visually impaired end user. 

The decoder is essentially the reverse of the encoder (148) and 
reads the digital signal previously encoded onto the unused line of video and 
reprocesses the digital stream using the original code. The decoder also 
converts the digital signal to an analog signal using a D to A converter. The 
signal is then routed through either the dedicated decoder box, existing 
television speakers, or external set of headphones for final listening through 
composite audio connector usually having a one volt peak to peak signal 
similar to the original audio signal. 
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The invention can have additional applications to include digital lines 
for foreign languages. The blind are also excluded form a critically valuable 
service: the on-screen typed messages of the emergency broadcast system, 
which does not include audio. The encoder/decoder device thus becomes 
the emergency broadcast system for the blind and visually impaired. Also, 
television program guides are now in a typed format on-screen for sighted 
people. Visually impaired are currently excluded from those types of 
program listings. 

Use of the present invention is beneficial as only the production 
facilities which create the master tapes would need to purchase the encoder 
(148) for implementation of the invention. With the increase of viewers, the 
producing company can acquire additional advertising dollars. Additionally, 
visually impaired-only audio advertising can be included in the narration 
audio signal so that products which are directed to the blind and visually 
impaired can be advertised directly to those consumers. This provides 
another potential source of income for the producer. Only the visually 
impaired and blind end users need purchase the decoding device or 
television, VCR, or other NTSC players which incorporate a decoding chip 
system. Thus, the cost of incorporating audio description is not born by 
those of normal sight nor of rebroadcasters, but rather those who derive 
benefit from the inclusion of audio description. 

Although several embodiments have been described in detail for 
purpose of illustration, various modifications may be made without departing 
from the scope and spirit of the invention. Accordingly, the invention is not 
to be limited, except as by the appended claims. 
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